Technical Field
This disclosure relates to image processing, and more specifically, to image processing in a camera system with multiple imaging channels.
Description of Related Art
Cameras with multiple imaging channels enable the extraction of distance information for objects in a captured image. For example, a captured image of a scene including a dog, a tree, and a cloud can be analyzed to determine a first distance from the camera to the dog, a second distance from the camera to the tree, and a third distance from the camera to the cloud. The distance between lenses in such cameras can create a parallax effect causing objects to appear at different positions within the images captured by each imaging channel. Calculating the position shifts of such objects between images enables the calculation of distance between objects in a scene.
When a multi-channel camera system captures an image, each channel captures a preliminary image of a scene from a slightly different perspective. The camera system then combines the preliminary images to form a final image. Prior to combining the preliminary images, the camera system modifies the preliminary images to ensure that any object in the preliminary images appears in the same location in each preliminary image. This modification of the preliminary images is referred to herein as “registration”, and requires 1) detecting and measuring object misalignment between preliminary images, and 2) applying a shift to misaligned objects (“object shift” hereinafter) in the preliminary images based on the measured misalignment.
Object misalignment between preliminary images can be measured using common image features (such as edges, patterns, and the like) detected in the preliminary images and comparing them until a shift is calculated. Relying on common image features to measure misalignment requires a particular feature to exist in at least two preliminary images. In an example camera system with three imaging channels (a “red channel” with a red filter, a “blue channel” with a blue filter, and a “green channel” with a green filter), an image feature can exist in one preliminary image, but not the other two. Such a problem can be solved by adding a redundant imaging channel (such as an extra green channel) to ensure that the image feature appears in at least two preliminary images.
Once a shift is calculated for a particular image feature associated with an object in a subset of the preliminary images, the distance of the object from the camera (“object distance” hereinafter) can be determined. Accordingly, since calculated object shifts during registration are dependent on the object distance, it is possible to calculate the shift that needs to be applied to the object in the remaining preliminary images. For example, in a four channel camera system having one red channel, one blue channel, and two green channels, an image feature can be identified in the preliminary images captured via the green channels. A shift can be calculated for the object associated with the image feature using the preliminary images captured by the green channels, and an object distance can be determined based on the calculated shift. When an object distance is known for the object associated with the image feature, a shift can be calculated and applied to the object in the preliminary images captured by the red and blue channels. The magnitude and direction of object shifts applied to each channel can vary based on the physical arrangement of the imaging lenses and the baseline between the each imaging channel's aperture.
In one embodiment, the multi-lens camera system described herein includes four lenses in a 2×2 array, one lens including a color filter associated with each of the colors red, green, blue, and white (also referred to as a “clear” filter). In such an embodiment, the white color filter includes higher light transmission characteristics, for instance including a broader passband and allowing a greater frequency range of light to pass through the filter. Accordingly, the white filter allows for the capture of preliminary images with higher signal-to-noise ratios and better low light performance. In addition, the inclusion of a white color filter enables the calculation of two sources of luminance: the signal from the white channel and the combined signals from the red, blue, and green channels. Such a configuration beneficially provides two luminance signals with different intensities that can be combined to form a combined luminance signal with a higher dynamic range than either individual luminance signal.
One challenge introduced with the use of three or more imaging channels, each including a different color filter is the possibility that a particular image feature may not appear in more than one preliminary image in view of the different spectra associated with each imaging channel. Accordingly, it is desirable to ensure that a shift can be calculated for any given image feature in a camera system with three or more color channels, each associated with a chromatically different filter.